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MORE THAN MEETS THE EYE: 

CURRICULAR AND PROGRAMMATIC EFFECTS 
ON STUDENT LEARNING 

Abstract 

This paper reports the effects of program characteristics and faculty activities on 
students’ experiences and, ultimately, the development of students’ analytical and group skills. 
Data come from nationally representative samples of 4,330 seniors, 1,243 faculty members, and 
147 engineering program chairs on 39 campuses nationwide. Findings indicate that program 
characteristics and faculty behaviors and values have significant, if relatively small and largely 
indirect, effects on student learning by encouraging (or discouraging) certain kinds of student 
experiences, which, in turn, influence student learning. The results point to a need for more 
complex designs than are typically adopted in most learning outcomes studies. 




MORE THAN MEETS THE EYE: 

CURRICULAR AND PROGRAMMATIC EFFECTS 
ON STUDENT LEARNING 

Background and Purpose 

After reviewing 30 years of research, Pascarella and Terenzini (1991, 2005) concluded 
that “the evidence strongly suggests that . . . multiple forces operate in multiple settings to 
influence student learning and change” (2005, p. 629). Their review indicated, however, that 
most studies of the effects of college on students adopt an overly narrow conceptual focus, 
concentrating on only a comparative handful of factors at a time. The result is a body of 
evidence that, with few exceptions (e.g., Astin, 1993), “present[s] only a partial picture of the 
forces at work” (2005, p. 630). 

Much of the research on college effects on students has focused primarily on some subset 
of students’ college experiences. These studies typically concentrate on the influences of the 
curriculum (e.g., the majors they choose and the courses they take), or the classroom (e.g., the 
kinds of pedagogies their instructors adopt), or students’ out-of-class experiences (e.g., where 
they live, co-curricular involvement, interactions with peers and faculty members, employment, 
family responsibilities). Relatively few studies, however, have examined the effects of students’ 
experiences in each of these venues comprehensively, assessing the effects of students’ 
experiences in each venue while controlling for their experiences in the others. The National 
Study of Student Learning (see Pascarella et al., 1996; Springer, Terenzini, Pascarella, & Nora, 
1995; Terenzini, Springer, Pascarella, & Nora, 1995) and the National Survey of Student 
Engagement (Kuh, 2001), two of the largest studies of college impacts undertaken in the past 15 
years rest on such a conceptual foundation and are noteworthy exceptions to the more restricted 
conceptual vision underlying most other studies. 
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Both the evidence and logic, however, suggest that factors in addition to students’ college 
experiences may influence their learning in ways not yet identified or fully understood. A body 
of research considerably smaller than that concerned with the “student experience” has examined 
institutional characteristics as explanatory factors. These “between-college effects” studies 
explore the extent to which student outcomes are attributable, net of other factors, to differences 
in the characteristics of the institutions students attend. Most of these studies rely primarily on 
structural features, such as type of control, curricular mission, size, and selectivity. Such 
measures, however, are consistently poor predictors of virtually any student outcome once 
entering students’ precollege characteristics are taken into account (see, for example, Astin, 

1993; Dey, et al., 1997; Pascarella & Terenzini, 1991, 2005). Occupational and economic 
outcomes are the sole exceptions to this generalization. Pascarella and Terenzini (1991, 2005) 
suggest that the conventional institutional descriptors are too remote from the student experience 
to have much impact and are, consequently, largely unrelated to gains in student learning. 

In efforts to move beyond an institution’s structural features as influences on student 
learning, some studies have examined college and universities’ internal organizational features 
as possible influences on student outcomes. For example, some scholars have explored 
differences in the allocation of financial resources as a possible source of institutional impacts on 
learning, or the effects of an institution’s financial context on degree completion (Titus, 2006). 
These studies suggest that institutional expenditure patterns can shape student experiences and 
outcomes, although the magnitude of the effects tend to be small (Belfield & Thomas, 2000; 
Ryan, 2004; Smart, Ethington, Riggs, & Thompson, 2002; Toutkoushian & Smart, 2001). Porter 
(2006) also identifies institutional structures (such as peer ability, institutional density, 
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differentiation in the curriculum, and an institution’s research orientation) that affect student 
engagement. 

Another (also relatively small) body of research identifies features of an institution’s 
operational functioning, climate, or culture that can be clustered to characterize internal 
organizational environments. Berger and Milem (2000), for example, identify five such 
environments: bureaucratic, collegial, political, symbolic, and systemic. These “organizational 
effects” studies suggest that organizational environments and cultures may be more influential 
forces than the more commonly used structural characteristics (e.g., Astin & Scherrei, 1980; 
Berger, 2000; 2001-2002, 2002; Braxton & Brier, 1989; Smart & Hamm, 1993; Berger & 

Milem, 2000 provide a thorough review of this literature). These environmental clusters are 
useful in explaining student outcomes, but they are relatively abstract, describing an 
organization’s general operational behaviors and tendencies to make decisions in a particular 
fashion. They resemble the conventional structural descriptors in their level of aggregation and 
generality and, consequently, are still relatively distal from students’ experiences and, for 
practical or policy purposes, resistant to intervention or change. 

It seems reasonable to suggest that other institutional features, such as peer environments, 
faculty cultures, and internal, structural, programmatic, and policy considerations are more 
proximal to students’ experiences and outcomes than factors considered to-date and may also 
play a role. These frequently overlooked factors may shape students’ experiences in subtle, if 
indirect ways, but their subtlety or indirectness should not be taken to mean they are 
inconsequential influences on student learning and development. Recent research suggests that 
the certain organizational contexts and internal features may influence student outcomes, but 
those influences are more likely to be indirect than direct, being mediated through the nature of 
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the peer environment and the kinds of student experiences they encourage or discourage 
(Lattuca, Terenzini, & Volkwein, 2005; Reason, Terenzini, & Domingo, 2005, 2006). 

In the mid-1990s, the Accreditation Board of Engineering Education (ABET) adopted 
criteria for evaluating engineering programs nationwide that were dramatically different from the 
previous criteria (Engineering Accreditation Commission, 1998). The new model, known as 
“EC2000” (shorthand for the title of the document specifying the new criteria), shifted the 
emphasis of the program accreditation review away from the possession of specific resources 
and facilities to 11 specific learning outcomes, such as “ designing] a system, component, or 
process to meet desired needs” and “functioning] on multidisciplinary teams.” In 2002, ABET 
commissioned a study of the effectiveness of EC2000 in increasing the level of engineering 
graduates’ preparation to enter the profession. The database developed for that assessment 
offered an exceptional opportunity to examine the proposition that what institutions do 
programmatically - their academic programs, practices, and policies, as well as the faculty 
culture, have a measurable, if indirect, effect on student learning. The study reported here 
undertook a comprehensive evaluation of the effects of internal organizational features, such as 
shifts in program curricula and characteristics, and changes in faculty members’ practices and 
values, on students’ learning-related experiences and, in turn, the development of their analytical 
and group skills. Although the subjects of this research were engineering students, programs, 
and faculty members, the findings (because of the relatively generic nature of the outcomes 
studied) have implications for enhancing learning in other professional and academic fields. 
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METHODS 



Conceptual Framework 

Figure 1 portrays the hypothesized quasi-causal sequence initiated by the introduction of 
the new EC2000 accreditation standards. The framework suggests that, if the new criteria are 
having an impact on undergraduate engineering education, those effects will be apparent first in 
changes in the curriculum and instructional practices engineering programs offer as faculty 
members move to align their curricula and instructional practices in ways likely to promote the 
learning outcomes specified in EC2000. One might also expect to see shifts in the faculty 
culture toward greater involvement in learner-centered activities such as outcomes assessment 
and professional development activities that might improve a faculty member’s teaching. One 
might anticipate changes in administrative policies and practices, such as increased emphasis on 
teaching and learning when making hiring, promotion, and tenure decisions. To the extent that 
these curricular, instructional, cultural, and organizational shifts are occurring, one might then 
expect the degree of those shifts to be reflected in differences in the nature and extent of the 
experiences students report. The framework, finally, suggests that differences in student 
experiences will be reflected in differences in student learning. 

— Insert Figure 1 About Here — 

This study evaluates the overall conceptual framework, ascertaining the extent to which 
effects of changes in program characteristics and faculty activities are reflected in differences in 
students’ experiences and, ultimately, in the development of their analytical and group- 
functioning skills. This set of propositions suggests that students’ development of these skills is 
a function not only of what occurs in the classroom and the kinds of out-of-class, engineering- 
related experiences students have during their undergraduate programs, but also of the 
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organizational components relating to program characteristics and faculty behaviors and values 
that shape those experiences. 

Design, Population, and Sample 

The analyses reported here are part of a large, three-and-one-half-year national study of 
the effects of the introduction of ABET’s new accreditation criteria on the preparation of 
undergraduate engineers to enter their profession. Because of time and resource constraints, the 
research group adopted a pre-/post-test, cross-sectional, ex post facto design that included 
graduates of the engineering classes of 1994 and 2004. Because the present study is concerned 
only with the effects of programmatic and cultural changes on student experiences and learning, 
the analyses here are based on data gathered in 2004. 

The population for the present study includes all 2004 graduates of ABET- accredited 
undergraduate engineering programs offering at least two of seven engineering disciplines: 
aerospace, chemical, civil, computer, electrical, industrial, and mechanical. This disciplinary 
array includes both those disciplines that produce the vast majority of engineering graduates in 
any given year (chemical, civil, electrical, and mechanical), as well as disciplines with strong ties 
to industry sectors (aerospace, computer, and industrial). 

The overall project dataset was developed using a two-stage, 7x3x2, disproportional, 
stratified random sample. Of the population of 1,241 ABET- accredited engineering programs in 
the targeted disciplines, 1,024 met the accredited-since-1990 specification. In the first stage, 40 
of those institutions were randomly selected within three strata: 1) the targeted seven disciplines, 
2) three categories in the EC2000 review cycle (i.e., programs that chose to be reviewed under 
EC2000 before being required to do so, to be reviewed when it was mandatory, or elected to 
defer EC2000 review when that option was available), and 3) whether the school’s programs had 




participated in a National Science Foundation Engineering Education Coalition during the 1990s 
(these coalitions were among the leaders in moving to meet the EC2000 criteria). To ensure an 
adequate number of responses for analysis within each discipline, selection was 
“disproportional” in over-sampling institutions with programs in the smaller disciplines 
(aerospace and industrial). To ensure representative samples of graduates and faculty members, 
four EC2000 pilot institutions (first reviewed in 1996 and 1997) were also included, as were 
several Historically Black Colleges and Universities (HBCUs) and Hispanic Serving Institutions 
(HSIs). The final sample included 147 programs at 39 institutions. 

Data Collection Procedures 

The overall project’s design entailed data collection from several sources, including 
graduating seniors in 2004, engineering graduates from the same programs in 1994, faculty 
members, program chairs, deans, and employers (data from deans, employers and the 1994 
graduates were not used in the study reported here). In the fall of 2003, survey instruments were 
sent to 2,971 faculty members and 203 program chairs. In spring, 2004, the population of 12,621 
seniors nearing graduation in any of the seven engineering fields on the each of the 40 campuses 
were sent the final survey instrument (all instruments are available at 
http://www.ed.psu.edu/cshe/abet/instruments.html) . One institution did not provide student 
responses, reducing the institutional sample to 39. The Graduating Senior Survey solicited 
information on basic demographic information, level of participation in out-of-class activities 
related to engineering education, student-learning outcomes associated with each of the 1 1 
EC2000 outcomes criteria, classroom practices, and plans for the future. Before data collection 
began, the dean of the college of engineering on each campus sent e-mail requests soliciting 
student and faculty participation. Follow-up waves included a postcard reminder sent two weeks 
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after the initial mailing, and a complete follow-up (similar to the initial mailing) sent two weeks 
after the postcard. 

Responses were received from 4,543 graduating seniors (a 36% response rate), 1,272 
faculty members, and the chairs of 147 engineering programs on 39 campuses. Any student or 
faculty case with more than 20 percent of the possible responses missing was deleted from the 
database, yielding 4,330 usable student cases (a usable response rate of 34%), 1,243 faculty 
members (a 42% response rate), and 147 (72% response) of the program chairs. Missing data in 
the usable student and faculty cases were imputed using expected maximization procedures 
(Allison, 2001). 

Graduating seniors described their undergraduate engineering experiences (both inside 
and outside the classroom) and reported their skill levels in engineering design and analysis and 
in working in groups. Program chairs reported on a wide array of program characteristics and 
curricular changes made since implementation of the EC2000 reaccreditation criteria. Faculty 
members reported on a broad array changes made in their instructional approaches and other 
professional practices and development activities over the past decade. 

Variables 

The criterion measures came from 36 items produced in a lengthy, detailed process to 
operationalize the 11 learning outcomes criteria specified in EC2000 (see Strauss & Terenzini, 
2005). Respondents were asked to indicate their level of achievement on each of these skill 
items using a 5-point scale, where 1 = “No Ability” and 5 = “High Ability.” A series of principal 
component analyses produced nine scales, two of which are the dependent variables in this study. 
Table 1 summarizes the composition of these two scales and their internal consistency 
(Cronbach’s alpha) reliabilities. The final, nine-factor solution retained 75.3 percent of the 
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overall item variance among the original 36 survey items, all of which loaded above .40 on a 
single factor; none loaded above .45 on any two factors (the full factor structure and scale 
reliabilities are given in Strauss & Terenzini, 2005). Factor scale scores were calculated by 
summing students’ responses to each item loading above .45 on a component and dividing by the 
number of items in the scale (Armor, 1974). Two of the nine factors, one dealing with students’ 
group skills, the other with their design and analytical skills were selected for this study. Scale 
alphas for these two scales are: Design and Analytical Skills = .92 and Group Skills = .86. These 
two scales were chosen for this study because each represents one of the two major categories 
frequently used to classify engineering skills: professional skills (such as communication skills, 
interpersonal skills) and technical skills (such as problem-solving, technological skills, and 
design skills). The Design and Analytical Skills scale taps students’ abilities to solve 
unstructured engineering problems (i.e., those for which there is no single or “correct” solution). 
The Group Skills scale reflects students’ abilities to work successfully with others to accomplish 
a team goal. 

— Insert Table 1 About Here — 

The independent variables fall into five sets: 1) students’ precollege characteristics (e.g., 
sociodemographic traits and academic preparation), 2) institutional characteristics and students’ 
discipline, 3) six scales measuring engineering program characteristics (e.g., curricular 
emphases); 4) nine scales tapping faculty members’ instructional practices and attitudes, as well 
as the faculty culture, and 5) ten items reflecting students’ in- and out-of-class activities relevant 
to engineering education. With three exceptions, the alpha reliabilities for all student, program, 
and faculty scales exceed .70. 



- 10 - 




Because the study was concerned with identifying the effects of changes in program 
curricula and pedagogies, the faculty culture, and program policies on students’ experiences and 
subsequent learning, potentially confounding precollege student characteristics were controlled, 
as were institutional characteristics and graduates’ engineering discipline. Individual differences 
used as covariates included students’ age, gender, high school preparation in basic math and 
science. Institutional characteristics include such differences as type of control, Carnegie 
classification, institutional size and wealth. The operational characteristics of these and all other 
variables are listed in Table 2. 

— Insert Table 2 About Here — 

Three scales represent students’ in-class experiences. The instructional practices set of 
variables included three, factorially derived (principal components with varimax rotation) scales: 
Clarity and Organization, Collaborative Learning, and Instructor Interaction and Feedback. Each 
scale was formed by summing its component items (loadings above .40) and dividing by the 
number of items in the scale. These scales consist of three, seven, and five items respectively, 
with alphas of .82, .90, and .87. Two scales were also created to represent the students’ 
perception of program openness to ideas and people. This scale consists of four indicators and 
has alphas of .74. There were also six out-of-class experiences measured through student 
responses. The first was program diversity climate which was a scale created from four 
indicators with an alpha of .57. The final five out-of-class experiences were conceptualized 
through single observed student responses (e.g., cooperative education experiences, study 
abroad, and internship/coop participation). The component items for each of these scales are 
given in Table 2. 
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Changes in the curriculum, pedagogies used, faculty culture, and program practices and 
policies are measured by nine faculty activities scales and six program scales. Changes in the 
program’s curriculum and organizational practices and policies were measured using six 
factorially derived (principal components with Varimax rotation) scales (created using 
procedures identical to those adopted to create the criterion measures). These scales tapped 
changes in the program’s curricular emphasis on content focusing on foundational engineering 
skills, contexts and standards for engineering, communication skills, and project-related skills. 
Two scales measured shifts in resources and in a program’s emphasis on teaching for hiring, 
promotion, and tenure decisions. 

Nine scales measured changes in faculty members’ instructional practices and attitudes. 
These indicators tapped increases in faculty members’ engagement in professional development 
and outcomes assessment projects to improve the education they and their programs offer, 
increased emphasis in their courses on engineering contexts (e.g., social, environmental, 
economic) and on ethics and professional standards, increased emphasis on key engineering 
project skills (e.g., communication and group skills), and any shifts in their courses’ emphasis on 
foundational engineering knowledge and applied skills. Two factors assessed faculty shifts in 
the learning pedagogies they employed. As with all other variables, the operational forms of 
these measures are given in Table 2. 

For analytical purposes, all faculty variables were converted to “composite” variables by 
averaging individual faculty members’ responses or scale scores within each program within 
each institution. These composite scores were then matched to individual students in the same 
engineering program at the same institution. Program variables derived from program chairs 
were similarly matched to individual students within the chairs’ programs. 
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Analytical Procedures 

Data were analyzed using hierarchical (or blocked) ordinary least-squares (OLS), 
multiple regression. Analyses proceeded in two phases. In Phase 1, each of the 10 student in- 
and out-of-class experience variables was regressed on students’ precollege characteristics (the 
control variables) and then on the program and faculty composite variables. In Phase 2, each of 
the two outcome scales was regressed on all control variables, all program and faculty composite 
variables, and all student experience variables. The beta weights (and changes in them as the 
analyses proceeded) provided information on direct and/or indirect effects of programmatic and 
faculty influences on students experiences and, ultimately, learning outcomes. 

In Phase 2, two regression models were run for each dependent variable. In both models, 
students’ precollege characteristics were entered as the first block. In the first regression, the 
program change and faculty activities variables were entered ahead of students’ in- and out-of- 
class experiences. In the second regression, the order was reversed, with the program- and 
faculty-change variables block entered after students’ in- and out-of-class experiences. This 
procedure was intended to allow estimation of the two unique-variance terms attributable to 
program and student-experience variable sets, as well as the joint contributions of the two sets. 
The joint contribution, in essence, is the residual variance explained after subtracting from the 
overall R" the variance due to students’ precollege characteristics and to the unique variance 
terms contributed by the program and student-experience variable sets. 

FINDINGS 

Analyses indicate that program curricula and structures and faculty practices and values 
shape students’ development of both their analytical and group skills. The programmatic and 
faculty influences, however, are largely indirect rather than direct, affecting student learning by 
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encouraging (or discouraging) certain kinds of student experiences, which, in turn, directly shape 
each student learning outcome. 

As can be seen in Table 3, and with Design and Analytic Skills as the criterion variable, 
the total variance explained by the model containing control variables, program changes, faculty 
activities, and student experiences is 23.2 percent. With Group Skills as the dependent variable, 
the model accounts for 18.1 percent of the total variance. For both outcome measures, however, 
the contributions of program-related changes are vastly smaller than those attributable to 
students’ experiences. The program changes and faculty activities uniquely explain less than one 
percent of the total variance. The joint variance explained by the two sets of variables is also 
very low. These findings indicate that the direct effects of program changes and faculty 
activities on either student learning outcome are not very powerful, although the fact that they 
explain any unique variance at all suggests the possibility that they may have indirect effects and, 
thus, are at least of theoretical interest. 

— Insert Table 3 About Here — 

Effects of Program Changes on Student Experiences. Evidence of broad (if modest) 
influence of program variables on students’ experiences, however, is apparent in Table 4 and 
consistent with theoretical expectations. All six of the program-related scales had a statistically 
significant influence (with p < .05 or lower) on at least one student experience, and three scales 
(increased emphasis on teaching in faculty personnel decisions, and increased curricular 
emphasis on communication skills and on project skills) had statistically significant effects (p < 
.05 or lower) on more than one student-experience. With the alpha significance criterion relaxed 
to p < .10 (not an unreasonable step, given the attenuated power of composite predictor variables 
and the interest in this study on testing a relatively new substantive proposition), five of the six 
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program variables predicted differences in two or more student experiences, and three program 
variables (changes in resources and increased emphasis on communication skills and on project 
skills) had an influence on three or more student experiences. Changes (mostly increases) in 
programmatic emphasis on communication skills and project skills appeared particularly 
influential. All of these influences are apparent even after taking into account students’ 
precollege characteristics and selected traits of the institutions they attend. For six of the ten 
student experiences, the faculty and program factors explained 10 percent or more of the 
variation in the student experiences, and two more approach that mark. 

— Insert Table 4 About Here — 

Effects of Faculty Changes on Student Experiences. The lower half of Table 4 
summarizes the evidence relating to faculty influences on students’ experiences. The findings 
shown there also suggest the possible presence of indirect effects and are consistent with the 
theoretical expectations implied in Figure 1 . All but one of the nine faculty variables have a 
statistically significant influence on at least one student experience (faculty involvement in 
innovations intended to improve engineering education is the sole exception). Four of the nine 
faculty activity shifts had a net effect on two student experiences, and one (an increased 
emphasis in class on project skills) had a significant and positive influence on four student 
experiences. Under the more forgiving p < .10 criterion, all nine faculty scales have an impact 
on at least one student experience, and four have an influence on three or more experiences. 
Several of these effects are rather robust, such as the positive influence of faculty members’ 
increased use of active learning pedagogies and increased course emphasis on foundational 
engineering content on students’ involvement in internships and cooperative education 
experiences, as well as the strong negative effect on that same experience of faculty emphasis on 
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applied skills. Some of these significant effects (by whatever standard) are negative and not 
easily interpretable, but the pattern clearly indicates a role for factors reflecting faculty practices 
and the faculty culture in the kinds of experiences graduates report. 

Effects of Student Experiences on Learning Outcomes. As can be seen in Table 5, and as 
one might expect, the measures of students’ in- and out-of-class experiences were by far the 
strongest predictors of the students’ group and design and analytical skills. Looking first at the 
Group Skills model, seven of the ten student experiences are significant, even after controlling 
for precollege student characteristics and institutional traits. Exposure to active and 
collaborative pedagogies had a positive, and the largest, effect on the level of students’ group 
skills. Students’ perceptions of their program’s openness to new ideas and people, and of their 
campus’ diversity climate, also had a noteworthy effect on the level of group skills the students 
developed. 

— Insert Table 5 About Here — 

Eight of the ten in- and out-of-class experiences were significant predictors of students’ 
Design and Analytical Skills. Their instructors’ clarity in the classroom and interactions with, 
and feedback to, students had the largest effects on students’ design and problem-solving skills. 

Effects of Program, Faculty, and Student Experiences on Learning. Table 6 summarizes 
the direct effects (beta weights) of all independent variables on students’ group and analytical 
skills after controlling for students’ precollege characteristics and institutional traits. Program 
characteristics and faculty activities account for only a small portion of the total variation in the 
outcome variables, but several scales reflecting changes faculty members reported making in 
their courses show statistically significant direct effects on the student outcome variables net of 
other factors. In programs where faculty members increased their emphasis in their courses on 



- 16 - 




foundational engineering knowledge (basic science, basic engineering science, and foundational 
math) and reported less emphasis on traditional pedagogies, students reported higher levels of 
learning on both outcomes than did their peers in programs with faculty members who devoted 
less course attention to foundational engineering content. Faculty perceptions that teaching was 
recognized favorably in the faculty hiring and reward system, however, appeared to affect those 
skills negatively, if only marginally. 

Students in programs where faculty members engaged in professional development 
activities to improve themselves as educators reported higher design and analytical skills than 
their counterparts in programs where faculty members were less engaged in teaching-related 
professional development activities. Only one program-related characteristic - increased 
emphasis on students’ communication skills - had a significant effect on a learning outcome, and 
its effects on student design and analytical skills was negative. No explanation for this finding is 
immediately apparent. 

— Insert Table 6 About Here — 

With the inclusion of program changes and faculty activity into the model, the student 
experience beta weights for all but one factor change at least somewhat. The most substantial 
change is the disappearance of students’ perceptions of their program’s diversity climate. The 
effects of participation in a student chapter of a professional society or association also 
disappears when predicting the Group Skills measure, and its statistical significance drops 
sharply when predicting students’ Design and Analytical Skills. Participation in engineering 
design competitions also loses power in influencing students’ group skills. Three other student 
experiences decline in influence with the addition of the program change and faculty activity 
factors. These drops in the effect sizes of student experiences when program and faculty factors 
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are taken into account constitutes additional evidence of the presence of indirect effects relating 
to program changes and faculty activities on learning outcomes through their influence on 
students’ experiences. These analyses would suggest that programmatic and faculty choices can 
have both direct and indirect effects on student learning outcomes. 

[.imitations 

This study, like any other, is limited in several ways. First, the dependent constructs - 
group skills and design and analytical abilities - are complex skill sets and not easily measured. 
Although the process used to create these scales closely followed the canons of survey 
instrument development (Strauss & Terenzini, 2005), the resulting measures may only partially 
reflect the range of talents needed to be proficient in these skill areas. Direct observations of 
these skills would be preferred but also difficult to obtain from the more than 4,000 students on 
39 campuses across the country who participated in the study. 

Second, only graduating seniors in engineering participated in the study. Although 
respondents were representative of the national population of undergraduate engineering 
graduates, generalization of study results to students in other academic areas (even in other 
science, technology, engineering, and math disciplines) is probably limited by the specialized 
and technical nature of engineering as a field of study. 

Third, the study relies on self-reports of student learning, and one must acknowledge the 
possible weaknesses of such data. A growing body of empirical evidence, however, suggests 
that, under certain conditions (believed to exist in this study’s outcome measures), self-reported 
learning can be a reasonable proxy for other, more objective measures (Anaya, 1999; Bradburn 
& Sudman, 1988; Carini, O’Day, & Kuh, 2002; Converse & Presser, 1989; Hayek, Laing, 
Sawyer, & Noble, 1988; Pace, 1985; Pike, 1995). Moreover, although self-reports have 
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acknowledged limitations when compared with standardized tests, the latter come with their own 
limitations, including availability, length, cost, administration requirements, and relevance to the 
outcomes of interest. 

Fourth, the evidence relating to the indirect effects of program characteristics and faculty 
behaviors and attitudes on student learning was not fully tested. Because of the complexity of 
the conceptual framework and number of predictor variables, no structural equation models were 
attempted. The findings reported here, however, suggest there may be information to gain from 
such analyses. 

CONCLUSIONS AND IMPLICATIONS 

This study examined the proposition that internal organizational structures, practices, and 
policies - what institutions do - affects learning. Assuming that multiple factors contribute to 
what and how students learn, we explored how changes in curricula, instruction, administrative 
policies and procedures, and faculty culture shaped students’ undergraduate educational 
experiences, and how, in turn, these changes influenced learning outcomes. The findings 
highlight the complexity of the processes at work and the importance of adopting comprehensive 
models to guide research on college students develop and change: Learning in engineering 
programs (and probably most other professional and academic fields) appears to be influenced 
not only by students’ experiences during college, but also by programmatic changes and faculty 
activities that encourage some kinds of experiences and discourage others. 

For two key engineering outcomes - group skills and design and analytical skills - what 
program faculty do and what they value makes a difference in student learning. Increased 
emphasis on course topics associated with the learning outcomes specified in the EC2000 
accreditation criteria, greater use of active learning and decreased use of traditional pedagogies, 
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greater faculty participation in professional development activities, and greater emphasis on 
teaching in the faculty reward system are consistently related to greater student involvement in 
learning-related experiences inside and outside the classroom. All but two of the faculty and 
program variables studied shaped at least one student experience, and a few influenced several. 
The effects of an emphasis on active learning techniques and foundational topics in math, 
science and engineering topics are fairly robust. The remaining effects tend to be small, 
although their small size may be attributable, at least in part, to the aggregation of data to the 
faculty average within programs and use of program-level data, with the consequent attenuation 
of variance in those predictors. The reported relations are, thus, conservative estimates of the 
unique contribution of the variables and the existence of any measurable effect is, as a result, 
worth noting. 

The study findings also indicate that students’ classroom experiences are significant, if 
modest, predictors of the two criterion measures. Collaborative experiences in the classroom, as 
well students’ perceptions that their academic program promotes diversity, tolerance, and respect 
for differences, contribute to the development of students’ group skills. Students’ in- and out-of- 
class experiences also moderately influence development of their design and analytical skills. 
Clarity of instruction, collaborative work in courses, and the amount of interaction with, and 
feedback from, faculty play a part in students’ development of skills in engineering design and 
analysis. 

Although less influential, out-of-class experiences such as internships, participation in 
design competitions, and involvement in student chapters of professional engineering 
organizations similarly make significant contributions to the outcomes under study. In addition, 
students’ perceptions of their programs’ openness to diverse ideas and people contribute to the 
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development of both group and design/analytical skills. The items in the program openness scale 
assess the degree to which students perceive their courses and program address and value an 
understanding of diversity. The more programs work to create a climate conducive to difference 
and discussions of difference, the more students develop skills that are essential to their success 
in the engineering profession. 

When all the variables are included in the regression models, all but one of the program 
scales drop out, but a number of faculty variables remain. Faculty members’ emphasis on 
foundational knowledge and decreased use of traditional pedagogies still have measurable, direct 
effects on the development of students’ group skills. In addition, faculty members’ participation 
in professional development activities focused on teaching and assessment, as well the emphasis 
they place in their courses on foundational knowledge, have direct effects on design and 
analytical skills. While student experiences are the largest and most consistent predictors of 
learning outcomes in the regression models, the persistence of these faculty-related direct effects 
strongly suggest that many factors in addition to students’ experiences promote learning in 
engineering programs. Moreover, the consistent pattern of indirect effects of program and 
faculty variables on learning via their influence on student experiences should alert researchers 
to be wary of the post hoc fallacy, that is, the assumption that because one event immediately 
precedes another, the first event causes or is the only influence on the subsequent variable. The 
evidence of indirect effects is a reminder not to overlook other, more remote and subtle, factors 
that also contribute to particular outcomes. These analyses show that many of the decisions that 
faculty make affect students’ educational experiences, which then in turn affect learning. 

In the quasi-causal sequence explored in this study, the pattern of direct and indirect 
relationships, and the particular factors that lead to the relationships between student experiences 



-21 - 




and learning outcomes, have implications for theory development, policy, and practice. 
Simplistic models of student learning imply that researchers can identify, and administrators and 
policy makers can adopt, silver bullets or best practices which will substantially improve student 
learning. More than two decades of research demonstrate that the equation is much more 
complex; many different factors affect student learning and development (Pascarella & 
Terenzini, 2005). Ignoring the indirect effects of programmatic factors presents an incomplete 
picture of the forces that shape student learning, reducing attention on what affects learning to a 
few variables that represent only a fraction of a student’s educational experience. Furthermore, 
the tendency to focus only on direct effects tends to obscure the role that program features and 
faculty activities such as curriculum revision and professional development play in promoting 
learning. In these analyses, faculty decisions about curricula and participation in professional 
development had significant direct effects on learning, but many additional program and faculty 
decisions also influenced the student experience inside and outside the classroom. Furthermore, 
institutional policies that fail to reward faculty for their efforts to enhance their pedagogical 
knowledge and program curricula ultimately shortchange students. Finally, the national policy 
conversation about student learning in higher education suffers when researchers and policy 
makers discount comprehensive research designs and reform efforts that acknowledge the 
contribution made by the many different links that constitute the chain of influences on student 
learning. The search for the silver bullet narrows, rather than expands, thinking and action. As 
long as such constricted views dominate academic and policy decision-making, educational 
reform and innovation will amount to tinkering at the margins, and the impacts likely to be 
predictably small. 
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Figure 1: Conceptual Framework for the Engineering Change Study 
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Table 1. Factor structure underlying items operationalizing two EC2000 learning outcomes 
criteria. 



Scale Label and Highest Loading Items 


Number 
of Items 


Factor 

Loadings 


Scale 

Alpha 


Percent 
of Total 
Variance 
Explained 


Design and Analytical Skills 


6 




.92 


12.4% 


Design solutions to meet desired needs 




.78 






Apply systematic design procedures to open-ended problems 




.77 






Define key engineering problems 




.76 






Formulate a range of solutions to an engineering problem 




.75 






Understand essential aspects of the engineering design process 




.66 






Apply discipline-specific engineering knowledge 




.47 






Group Skills 


3 




.86 


7.1 


Work with others to accomplish team goals 




.86 






Work in teams of people with a variety of skills and backgrounds 




.84 






Work in teams where knowledge and ideas from multiple 
engineering disciplines must be applied 




.67 






TOTAL VARIANCE EXPLAINED BY 9-FACTOR 
SOLUTION 








75.3% 
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Table 2. Independent variables used in OLS regression analyses. 



Control Variables 

1. Students’ Preeollege Characteristics 

Age: Actual years 
Sex: 1 = male, 0 = female 

Transfer status: 1 = native freshman on entry, 0 = transfer 
Citizenship: 1 = U.S. citizen, 0 = not a U.S. citizen 

Race/Ethnicity: Black/ African American, Hispanic or Latino, Asian, American 
Indian/ Alaskan Native, Hawaiian or Pacific Islander, and other, each coded as a dummy 
variable (1 or 0) with white used as the comparison group 

Family income: 9-point scale, where 1 = below $20,000 and 9 = more than $150,000 

Highest level of education attained by mother: 4-point scale, where 1 = high school diploma, 
GED, or less and 4 = advanced degree 

Highest level of education attained by father: 4-point scale, where 1 = high school diploma, 
GED, or less and 4 = advanced degree 

SAT scores: Actual scores on both the math and verbal sections of the SATs 

Overall high school GPA: 6-point scale, where 1 = below 1.49 (below C-) and 6 = 3.5 to 4.0 
(A to A-) 

Preparation for basic science and math courses when entering college: An Individual’s 
response to the question: How well prepared were you for basic science and math courses 
when you entered college? On a 4-point scale, where 1 = not at all and 4 = very well 
prepared 

2. Institutional Characteristics 

Type of control: 1 = public, 0 = private 

NSF coalition participation: 1 = member of coalition, 0 = not a coalition member 

EC2000 review schedule: early (1998-2000) and on-time (2001-2003) - each coded as a 
dummy variable (1 or 0) with late (2004-2006) used as the comparison group 

Carnegie classification: Carnegie Research Extensive, Carnegie Research Intensive, and 
Carnegie Masters - each coded into dummy variables (1 or 0) with Carnegie Bachelors as the 
comparison group 

Wealth: Average salary of full professors in engineering, converted to z-scores 

Size: Number of undergraduate engineering degrees awarded in 2004, converted to z-scores 

Engineering discipline: Industrial, Mechanical, Aerospace, Chemical, Civil, and Computer - 
each coded as dummy variables (1 or 0) with Electrical used as the comparison group 
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Independent Variables 



3. Program Changes 

Emphasis on Teaching in Faculty Personnel Decisions : The program chair’s individual score 
on a 3-item scale (where 5 = significant increase, and 1 = significant decrease) that assesses 
the change in the emphasis on teaching in personnel matters. Constituent items: “Recruiting 
and hiring;” “Promotion and tenure;” and “Salary and merit increases.” (Alpha = .89) 

Change in Resources'. The program chair’s individual score on an 8-item scale (where 5 = 
significant increase and 1 = significant decrease) that assesses the approximate change in 
program resources over the past 10 years. Constituent items: “Instructional staff;” “Support 
staff;” “Instructional resources;” “Institutional funds for innovations;” “Faculty professional 
development;” “Research resources;” “Facilities;” “Financial resources.” (Alpha = .87) 

Emphasis on Foundational Knowledge: The program chair’s individual score on a 5-item 
scale (5 = significant increase, to 1 = significant decrease) that assesses the approximate 
change in the program's emphasis on the following areas over the past decade. Constituent 
items: “Basic engineering science;” “Foundational math;” “Basic science;” “Experimental 
methods;” “Engineering problem solving.” (Alpha = .74) 

Contexts and Standards: The program chair’s individual score on a 5-item scale (5 = 
significant increase, to 1 = significant decrease) that assesses the approximate change in the 
program's emphasis on the following areas over the past decade. Constituent items: 
“Professional ethics;” “Engineering in global/social contexts;” “Professional responsibility;” 
“Life-long learning;” “Knowledge of contemporary issues.” (Alpha = .79) 

Communication Skills: The program chair’s individual score on a 3-item scale (where 5 = 
significant increase, to 1 = significant decrease) that assesses the approximate change in the 
program's emphasis on the following areas over the past decade. Constituent items: “Verbal 
communication;” “Technical writing;” “Interpersonal/group communication.” (Alpha = .79) 

Project Skills: The program chair’s individual score on a 4-item scale (where 5 = significant 
increase, to 1 = significant decrease) that assesses the approximate change in the program's 
emphasis on the following areas over the past decade. Constituent items: “Engineering 
design;” “Use of modem engineering tools;” “Project management;” “Teamwork.” (Alpha = 
.72) 

4. Instructional Practices and Attitudes of Faculty Members 

Personal Improvement: An average of all the individual faculty member’s scores in each 
program. Each individual faculty member score is a 4-item scale (where 5 = significant 
increase, to 1 = significant decrease) that asses the faculty member’s participation compared 
to five years ago. Constituent items: “Seminars or workshops on assessing students 
learning;” “Seminars or workshops on teaching and learning;” “Using services of on-campus 
instructional center;” “Reading materials on teaching.” (Alpha = .73) 

Projects to improve engineering education : An average of all the individual faculty 
member’s scores in each program. Each individual faculty member score is a 5-item scale 
(where 5 = significant increase, to 1 = significant decrease) that asses the faculty member’s 
participation compared to five years ago. Constituent items: “A project to improve 
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undergraduate engineering education;” “Applying for external funding for an undergraduate 
engineering education project;” “Developing or teaching a course with someone in another 
engineering discipline;” “Conference or journal submission on undergraduate education;” 
“Activities to enhance content knowledge.” (Alpha = .64) 

Contexts and Standards: An average of all the individual faculty member’s scores in each 
program. Each individual faculty member score is a 4-item scale (5 = significant increase, to 
1 = significant decrease) that asses the approximate change in the emphasis on the following 
areas since the first time they taught the same course. Constituent items: “Professional 
ethics;” “Engineering in global/social contexts;” “Professional responsibility;” “Knowledge 
of contemporary issues.” (Alpha = .78) 

Emphasis on Foundational Knowledge : An average of all the individual faculty member’s 
scores in each program. Each individual faculty member score is a 3-item scale (where 5 = 
significant increase, to 1 = significant decrease) that asses the approximate change in the 
emphasis on the following areas since the first time they taught the same course. Constituent 
items: “Basic science;” “Basic engineering science;” “Foundational math.” (Alpha = .84) 

Project Skills: An average of all the individual faculty member’s scores in each program. 
Each individual faculty member score is a 4-item scale (where 5 = significant increase, to 1 = 
significant decrease) that asses the approximate change in the emphasis on the following 
areas since the first time they taught the same course. Constituent items: “Verbal 
communication;” “Technical writing;” “Teamwork;” “Project management.” (Alpha = .74) 

Applied Skills: An average of all the individual faculty member’s scores in each program. 
Each individual faculty member score is a 3-item scale (where 5 = significant increase, to 1 = 
significant decrease) that asses the approximate change in the emphasis on the following 
areas since the first time they taught the same course. Constituent items: “Modern 
engineering tools;” “Experimental methods;” “Engineering design.” (Alpha = .51) 

Active Learning Pedagogies: An average of all the individual faculty member’s scores in 
each program. Each individual faculty member score is a 7-item scale (where 5 = significant 
increase, to 1 = significant decrease) that asses the approximate change in the emphasis on 
the following areas since the first time they taught the same course. Constituent items: 
“Design projects;” “Assignments or exercises focusing on application;” “Open-ended 
problems;” “Student presentations;” “Use of groups in class;” “Hands-on experiences;” 

“Case studies or real-world examples.” (Alpha = .78) 

Traditional Pedagogies Scale: An average of all the individual faculty member’s scores in 
each program. Each individual faculty member score is a 2-item scale (where 5 = significant 
increase, to 1 = significant decrease) that asses the approximate change in the emphasis on 
the following areas since the first time they taught the same course. Constituent items: 
“Lectures;” “Problems form the textbook.” (Alpha = .48) 

Emphasis on Teaching in Faculty Personnel Decisions: An average of all the individual 
faculty member’s scores in each program. Each individual faculty member score is a 3-item 
scale (where 5 = significant increase, to 1 = significant decrease) that asses the approximate 
change in the emphasis on the following areas since the first time they taught the same 
course. Constituent items: “Recruiting and hiring;” “Promotion and tenure;” “Salary and 
merit increases.” (Alpha = .90) 
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5. Student reports of their in-class and out-of-class activities relevant to engineering 

Clarity and Organization Scale : An individual student’s score on a 3-item scale (where 4 = 
almost always, to 1 = almost never) assessing how often things happened in their classes. 
Constituent items: “Assignments and class activities were clearly explained;” “Assignments, 
presentations, and learning activities were clearly related to one another;” “Instructors made 
clear what was expected of students in the way of activities and effort.” (Alpha = .82) 

Collaborative Learning Scale: An individual student’s score on a 7-item scale (where 4 = 
almost always, to 1 = almost never) assessing how often things happened in their classes. 
Constituent items: “I worked cooperatively with other students on course assignments;” 
“Students taught and learned from each other;” “We worked in groups;” “I discussed ideas 
with my classmates (individuals or groups);” “I got feedback on my work or ideas from my 
classmates;” “I interacted with other students in the course outside of class;” “We did things 
that required students to be active participants in the teaching and learning process.” (Alpha 
= .90) 

Instructor Interaction and Feedback Scale : An individual student’s score on a 5-item scale 
(where 4 = almost always, to 1 = almost never) assessing how often things happened in their 
classes. Constituent items: “Instructors gave me frequent feedback on my work;” “Instructors 
gave me detailed feedback on my work;” “Instructors guided students’ learning activities 
rather than lecturing or demonstrating the course material;” “I interacted with instructors as 
part of the course;” “I interacted with instructors outside of class (including office hours, 
advising, socializing, etc.).” (Alpha = .87) 

Program Encouragement for Openness : An individual student’s score on a 4-item scale 
(where 4 = almost always, to 1 = almost never) assessing how often things happened in the 
program both in-class and out-of-class. Constituent items: “My engineering courses 
encouraged me to examine my beliefs and values;” “My engineering courses emphasized 
tolerance and respect for differences;” “My department emphasizes the importance of 
diversity in the engineering workplace;” “My engineering friends and I discussed diversity 
issues.” (Alpha = .74) 

Perceived Program Climate : An individual student’s score on a 4-item scale assessing how 
often things happened in the program when out-of-class. Constituent items: “In my major, I 
observed the use of offensive words, behaviors, or gestures directed at students because of 
their identity” (5 = strongly disagree, to 1 = strongly agree); “I was harassed or hassled by 
others in my major because of my identity” (5 = strongly disagree, to 1 = strongly agree); “I 
know some students who feel like they don't fit in this department because of their identity” 
(5 = strongly disagree, to 1 = strongly agree); “The faculty in my department are committed 
to treating all students fairly(5 = strongly agree, to 1 = strongly disagree).” (Alpha = .57) 

Internship/Coop Experience: 5-point scale, measuring the months as an intern or cooperative 
education student, where 1 = none and 5 = more than 12 months 

Study Abroad: 5-point scale, measuring the months in a study abroad program, where 
1 = none and 5 = more than 12 months 

International Travel: 5-point scale, measuring the months spent traveling internationally (not 
study abroad), where 1 = none and 5 = more than 12 months 
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Participation in Design Competition : 5-point scale, measuring the months spent in student 
design projects beyond classroom requirements, where 1 = none and 5 = more than 12 
months 

Involvement in Professional Society Chapter: 4-point scale, measuring the level of activity in 
a student chapter of a professional organization, where 1 = not at all and 4 = highly 
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Table 3. Partitioning of variance results for Design and Analytical Skills and Group Skills 
scales. 



Design & Analytical 
Skills 


Group Skills 


Variance due to precollege and institutional 






characteristics 


.089*** 


042*** 


Unique variance 3 due to: 






Program Changes and Faculty Activities 


.008* 


.008* 


Student Experiences 


132*** 


125*** 


Shared Variance 


.003 b 


.006 b 


Total Variance Explained 


232*** 


Igl*** 


Adjusted R 2 


216*** 


164*** 



Variance unattributable to any other variable set in the model. 
b Cannot be tested for statistical significance. 

*p< .05, ***p < .001 
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Table 4. Effects 1 of program changes and faculty activities on students experiences. 



Prof 





Clarity 


Collaboration Interaction 


Program 

Openness 


Climate 


Internship/ 

Coop 


Study 

Abroad 


Travel 


Design 

Competition 


Soc 

Chp 


Program Scales 

Emphasis on Teaching in 

Faculty Personnel Decisions 






-.050* 














Change in Resources 




.027+ .026+ 






-.051+ 


.023* 








Emphasis on 

Foundational Knowledge 




-.079+ 




.078* 












Contexts and Standards 




.086* 
















Communication Skills 






.092+ 








.088* 


.205** 




Project Skills 








-.047+ 


.150* 


-.040+ 


-. 122 *** 




.114+ 


Faculty Scales 

Personal Improvement 


















.518* 


Projects to Improve 

Engineering Education 














.189+ 






Contexts and Standards 


-.104+ 




-.126+ 


-.117* 




-.105* 








Project Skills 




.190** 


.171* 


.131* 




.151** 








Emphasis on 

Foundational Knowledge 










.346* 










Applied Skills 




-.157* 






-.705*** 










Active Learning Pedagogies 


.138* 








494 ** 


-.098+ 




.250+ 




Traditional Pedagogies 










.263* 






-.175+ 




Emphasis on Teaching in 

Faculty Personnel Decisions 


-.048+ 


-.078* 




-.061* 






-.038+ 




-.117+ 


R 2 


.100 


.123 .139 


.111 


.096 


.106 


.065 


.147 


.054 


.089 


+ p < .10, * p < .05, ** p < .01, *** 


p < .001 



















a Adjusted for student transfer status, age, gender, U S citizenship, race/ethnicity, mother's education, father's education, family income, test scores, high school 
GPA, discipline, and for institutional control (public/private), NSF coalition participation, EC2000 review schedule, Carnegie Classification, wealth, and size. 
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Table 5. Effects a of student experiences on Design and Analytical Skills and Group Skills 
scales. 



Student Experiences 


Group 

Skills 


Design & 
Analytical Skills 


In-Class 


Clarity 


.066*** 


.149*** 


Collaboration 


.275*** 


.059*** 


Instructor Interaction and Feedback 




.136*** 


Program Openness 


.101*** 


.089*** 


Out-of-Class 


Climate 


.073*** 


.084*** 


Internship/Coop Experience 


.026*** 


.029*** 


Study Abroad 
International Travel 


Participation in Design Competition 


.045*** 


.082*** 


Involvement in Professional Society Chapter 


.024** 


.024** 


R 2 


.167 


.224 



+ p < .10, * p < .05, ** p < .01, *** p < .001 



Adjusted for student transfer status, age, gender, U S citizenship, race/ethnicity, mother's education, father's 
education, family income, test scores, high school GPA, discipline, and for institutional control (public/private), 
NSF coalition participation, EC2000 review schedule, Carnegie Classification, wealth, and size. 
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Table 6. Effects a of program changes and faculty activities and student experiences on 
Design and Analytical Skills and Group Skills scales. 





Group Skills 


Design & 
Analytical 
Skills 


Program Characteristics and Faculty Activities 
Program Scales 

Emphasis on Teaching in Faculty Personnel 

Decisions 

Change Resources 

Emphasis on Foundational Knowledge 
Contexts and Standards 
Communication Skills 




i 

o 

* 


Project Skills 
Faculty Scales 

Personal Improvement 




.196* 


Projects to Improve Engineering Education 
Contexts and Standards 
Project Skills 

Emphasis on Foundational Knowledge 


242*** 


.159* 


Applied Skills 

Active Learning Pedagogies 

Traditional Pedagogies 


-.154** 




Emphasis on Teaching in Faculty Personnel 
Decisions 


-.066* 




Student Experiences 
In-Class 

Clarity 




.186*** 


Collaboration 


225*** 


.052** 


Instructor Interaction and Feedback 




.120*** 


Program Openness 


.130*** 


125*** 


Out-of-Class 

Climate 

Intemship/Coop Experience 


.031*** 


032*** 


Study Abroad 

International Travel 

Participation in Design Competition 


.036** 


.089*** 


Involvement in Professional Society Chapter 




.018+ 


R 2 


.181 


.232 



+ p < .10, * p < .05, ** p < .01, *** p < .001 

Adjusted for student transfer status, age, gender, U S citizenship, race/ethnicity, mother's education, father's 
education, family income, test scores, high school GPA, discipline, and for institutional control (public/private), 
NSF coalition participation, EC2000 review schedule, Carnegie Classification, wealth, and size. 
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